According to the problem that the existing roughness descriptors are mostly dependent on the average grey value, which is easy to cause the loss of image information, a new roughness descriptor based on Gaussian scale space was presented for pollen image classification and recognition. With this method, the Gaussian pyramid algorithm was used to divide the image into several different levels of scale space, and then the roughness texture feature was extracted from the different level scale space. The statistical distribution of roughness frequency was calculated to build the Scale-Space Roughness Histogram Descriptor (SSRHD). At last, the Euclidean distance was used to measure the similarity between images. The simulation results on Confocal and Pollenmonitor image database demonstrate that, compared with Discrete Hidden Markov Model Descriptors (DHMMD), the Correct Recognition Rate (CRR) performed by the SSRHD increases by 2.32% on Confocal and 1.2% on Pollenmonitor, and the False Recognition Rate (FRR) decreases by 0.1% on Confocal. The experimental results show that the SSRHD feature can effectively describe the pollen image texture and it also has good robustness to pollen rotation and pose variation.